The Democratization of Big Data

In recent years, it has become common for discussions about managing and analyzing information to reference “data scientists” using “the cloud” to analyze “big data.” Indeed these terms have become so ubiquitous in discussions of data processing that they are covered in popular comic strips like Dilbert and the terms are tracked on Gartner’s Hype cycle. The Harvard Business Review even labeled data scientist as “the sexiest job of the 21st century.” The goal of this paper is to demystify these terms and, in doing so, provide a sound technical basis for exploring the policy challenges of analyzing large stores of information for national security purposes. It is worth beginning by proposing a working definition for these terms before exploring them in more detail. One can spend much time and effort developing firm definitions for these terms – it took the National Institutes of Science and technology several years and sixteen versions to build consensus around the definition of cloud computing in NIST Special Publication 800-145 – the purpose here is to provide definitions that will be useful in furthering discussions of policy implications. Rather than defining big data in absolute terms (a task made nearly impossible by the rapid pace of advancements in computing technologies) one can define big data as a collection of data that is so large that it exceeds one’s capacity to process it in an acceptable amount of time with available tools. This difficulty in processing can be a result of the data’s volume (e.g., its size as measured in petabytes), its velocity (e.g., the number of new data elements added each second), or its variety (e.g., the mix of different types of data including structured and unstructured text, images, videos, etc . . . ). Examples abound in the commercial and scientific arenas of systems managing massive quantities of data. YouTube users upload over one hundred hours of video every minute, Wal-Mart processes more than one million transactions each hour, and Facebook stores, accesses and analyzes more than thirty petabytes